from sklearn_benchmarks.reporting.hp_match import HPMatchReporting

scikit-learn vs. ONNX Runtime

reporting = HPMatchReporting(against_lib="onnx", config="config.yml")
reporting.make_report()

We assume here there is a perfect match between the hyperparameters of both librairies. For a given set of parameters and a given dataset, we compute the speedup time scikit-learn / time onnx. For instance, a speedup of 2 means that onnx is twice as fast as scikit-learn for a given set of parameters and a given dataset.

KNeighborsClassifier_brute_force

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: algorithm=brute.

WARNING! Mismatch between validation scores for 12 predictions (75.0%). See below.

	estimator	function	n_samples_train	n_samples	n_features	accuracy_score_sklearn	accuracy_score_onnx	diff_accuracy_scores
1	KNeighborsClassifier_brute_force	predict	100000	1000	100	0.663	1.000	0.337
2	KNeighborsClassifier_brute_force	predict	100000	1	100	1.000	0.757	0.243
4	KNeighborsClassifier_brute_force	predict	100000	1000	100	0.757	0.882	0.125
7	KNeighborsClassifier_brute_force	predict	100000	1000	100	0.882	1.000	0.118
8	KNeighborsClassifier_brute_force	predict	100000	1	100	1.000	0.757	0.243
10	KNeighborsClassifier_brute_force	predict	100000	1000	100	0.882	0.663	0.219
13	KNeighborsClassifier_brute_force	predict	100000	1000	100	0.757	1.000	0.243
14	KNeighborsClassifier_brute_force	predict	100000	1	100	1.000	0.922	0.078
16	KNeighborsClassifier_brute_force	predict	100000	1000	100	0.663	0.929	0.266
19	KNeighborsClassifier_brute_force	predict	100000	1000	2	0.896	1.000	0.104
20	KNeighborsClassifier_brute_force	predict	100000	1	2	1.000	0.922	0.078
22	KNeighborsClassifier_brute_force	predict	100000	1000	2	0.922	0.896	0.026

Raw results

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.012	0.000	6.631	0.0	-1	1	17.024	0.126	0.663	0.001	0.001	See	See
3	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.012	0.000	6.524	0.0	-1	5	0.316	0.004	1.000	0.039	0.039	See	See
6	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.012	0.000	6.779	0.0	1	100	16.930	0.024	0.882	0.001	0.001	See	See
9	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.013	0.000	6.380	0.0	-1	100	0.318	0.005	1.000	0.039	0.039	See	See
12	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.012	0.000	6.508	0.0	1	5	3.702	0.006	0.896	0.003	0.003	See	See
15	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.013	0.000	6.365	0.0	1	1	0.239	0.004	1.000	0.053	0.053	See	See
18	KNeighborsClassifier_brute_force	fit	100000	100000	2	0.005	0.001	0.319	0.0	-1	1	3.715	0.018	0.929	0.001	0.001	See	See
21	KNeighborsClassifier_brute_force	fit	100000	100000	2	0.004	0.000	0.380	0.0	-1	5	0.243	0.004	1.000	0.017	0.017	See	See

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	KNeighborsClassifier_brute_force	predict	100000	1000	100	1.904	0.125	0.000	0.002	-1	1	0.311	0.002	1.000	6.128	6.128	See	See
2	KNeighborsClassifier_brute_force	predict	100000	1	100	0.026	0.003	0.000	0.026	-1	1	16.818	0.008	0.757	0.002	0.002	See	See
4	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.694	0.026	0.000	0.003	-1	5	16.877	0.026	0.882	0.160	0.160	See	See
5	KNeighborsClassifier_brute_force	predict	100000	1	100	0.027	0.003	0.000	0.027	-1	5	0.317	0.005	1.000	0.084	0.084	See	See
7	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.058	0.003	0.000	0.002	1	100	0.319	0.006	1.000	6.446	6.447	See	See
8	KNeighborsClassifier_brute_force	predict	100000	1	100	0.024	0.000	0.000	0.024	1	100	17.030	0.200	0.757	0.001	0.001	See	See
10	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.720	0.040	0.000	0.003	-1	100	17.286	0.034	0.663	0.157	0.157	See	See
11	KNeighborsClassifier_brute_force	predict	100000	1	100	0.027	0.002	0.000	0.027	-1	100	0.306	0.011	1.000	0.087	0.087	See	See
13	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.053	0.004	0.000	0.002	1	5	0.240	0.005	1.000	8.548	8.550	See	See
14	KNeighborsClassifier_brute_force	predict	100000	1	100	0.024	0.001	0.000	0.024	1	5	3.628	0.008	0.922	0.007	0.007	See	See
16	KNeighborsClassifier_brute_force	predict	100000	1000	100	1.211	0.008	0.001	0.001	1	1	3.698	0.007	0.929	0.328	0.328	See	See
17	KNeighborsClassifier_brute_force	predict	100000	1	100	0.025	0.001	0.000	0.025	1	1	0.241	0.004	1.000	0.102	0.102	See	See
19	KNeighborsClassifier_brute_force	predict	100000	1000	2	1.752	0.022	0.000	0.002	-1	1	0.239	0.005	1.000	7.325	7.326	See	See
20	KNeighborsClassifier_brute_force	predict	100000	1	2	0.005	0.004	0.000	0.005	-1	1	3.655	0.011	0.922	0.001	0.001	See	See
22	KNeighborsClassifier_brute_force	predict	100000	1000	2	2.642	0.039	0.000	0.003	-1	5	3.695	0.005	0.896	0.715	0.715	See	See
23	KNeighborsClassifier_brute_force	predict	100000	1	2	0.005	0.000	0.000	0.005	-1	5	0.241	0.005	1.000	0.020	0.020	See	See

KNeighborsClassifier_kd_tree

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: algorithm=kd_tree.

WARNING! Mismatch between validation scores for 12 predictions (75.0%). See below.

	estimator	function	n_samples_train	n_samples	n_features	accuracy_score_sklearn	accuracy_score_onnx	diff_accuracy_scores
1	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.929	1.000	0.071
2	KNeighborsClassifier_kd_tree	predict	1000000	1	10	1.000	0.946	0.054
4	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.946	0.951	0.005
7	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.951	1.000	0.049
8	KNeighborsClassifier_kd_tree	predict	1000000	1	10	1.000	0.946	0.054
10	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.951	0.929	0.022
13	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.946	1.000	0.054
14	KNeighborsClassifier_kd_tree	predict	1000000	1	10	1.000	0.911	0.089
16	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.929	0.894	0.035
19	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.891	1.000	0.109
20	KNeighborsClassifier_kd_tree	predict	1000	1	2	1.000	0.911	0.089
22	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.911	0.891	0.020

Raw results

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.066	0.076	0.026	0.0	-1	1	109.435	0.000	0.929	0.028	0.028	See	See
3	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.665	0.028	0.022	0.0	-1	5	2.598	0.163	1.000	1.411	1.413	See	See
6	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.637	0.067	0.022	0.0	1	100	108.694	0.000	0.951	0.033	0.033	See	See
9	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.605	0.093	0.022	0.0	-1	100	2.604	0.141	1.000	1.385	1.387	See	See
12	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.669	0.067	0.022	0.0	1	5	0.039	0.001	0.891	95.041	95.053	See	See
15	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.562	0.050	0.022	0.0	1	1	0.005	0.000	1.000	696.647	696.812	See	See
18	KNeighborsClassifier_kd_tree	fit	1000	1000	2	0.001	0.001	0.021	0.0	-1	1	0.061	0.001	0.894	0.013	0.013	See	See
21	KNeighborsClassifier_kd_tree	fit	1000	1000	2	0.001	0.000	0.031	0.0	-1	5	0.005	0.000	1.000	0.101	0.101	See	See

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.798	0.983	0.000	0.001	-1	1	2.603	0.191	1.000	0.306	0.307	See	See
2	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.002	0.001	0.000	0.002	-1	1	107.641	0.000	0.946	0.000	0.000	See	See
4	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.996	0.300	0.000	0.001	-1	5	108.643	0.000	0.951	0.009	0.009	See	See
5	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.003	0.001	0.000	0.003	-1	5	2.637	0.217	1.000	0.001	0.001	See	See
7	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	5.480	0.267	0.000	0.005	1	100	2.621	0.125	1.000	2.091	2.093	See	See
8	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.003	0.001	0.000	0.003	1	100	108.874	0.000	0.946	0.000	0.000	See	See
10	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	3.125	0.253	0.000	0.003	-1	100	109.597	0.000	0.929	0.029	0.029	See	See
11	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.005	0.001	0.000	0.005	-1	100	2.623	0.145	1.000	0.002	0.002	See	See
13	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	1.649	0.172	0.000	0.002	1	5	0.005	0.000	1.000	323.731	323.823	See	See
14	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.002	0.000	0.000	0.002	1	5	0.040	0.002	0.911	0.040	0.040	See	See
16	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.936	0.237	0.000	0.001	1	1	0.060	0.000	0.894	15.513	15.513	See	See
17	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.001	0.000	0.000	0.001	1	1	0.005	0.000	1.000	0.200	0.200	See	See
19	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.025	0.011	0.001	0.000	-1	1	0.005	0.000	1.000	4.899	4.901	See	See
20	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.002	0.000	0.000	0.002	-1	1	0.040	0.001	0.911	0.051	0.051	See	See
22	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.023	0.001	0.001	0.000	-1	5	0.039	0.000	0.891	0.595	0.595	See	See
23	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.002	0.000	0.000	0.002	-1	5	0.005	0.000	1.000	0.393	0.393	See	See

HistGradientBoostingClassifier_best

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: learning_rate=0.01, n_iter_no_change=10.0, max_leaf_nodes=100.0, max_bins=255.0, min_samples_leaf=100.0, max_iter=300.0.

WARNING! Mismatch between validation scores for 1 prediction (100.0%). See below.

	estimator	function	n_samples_train	n_samples	n_features	accuracy_score_sklearn	accuracy_score_onnx	diff_accuracy_scores
1	HistGradientBoostingClassifier_best	predict	100000	1000	100	0.824	1.0	0.176

Raw results

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	HistGradientBoostingClassifier_best	fit	100000	100000	100	95.382	0.0	300	0.001	0.001	0.496	0.021	0.824	192.191	192.364	See	See

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	HistGradientBoostingClassifier_best	predict	100000	1000	100	0.107	0.001	300	0.008	0.0	0.411	0.007	1.0	0.26	0.26	See	See